Adaptive Orthogonal Projection for Batch and Online Continual Learning

Catastrophic forgetting is a key obstacle to continual learning. One of the state-of-the-art approaches orthogonal projection. The idea this approach learn each task by updating network parameters or weights only in direction subspace spanned all previous inputs. This ensures no interference with tasks that have been learned. system OWM uses performs very well against other systems. In paper, we first discuss an issue discovered mathematical derivation and then propose novel method, called AOP (Adaptive Orthogonal Projection), resolve it, which results significant accuracy gains empirical evaluations both batch online learning settings without saving any training data as replay-based methods.